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Abstract 

We consider the problem of Linear Programming (LP) decoding of binary linear codes. The LP 
excess lemma was introduced by the first author, B. Ghazi, and R. Urbanke (IEEE Trans. Inf. Th., 2014) 
as a technique to trade crossover probability for “LP excess” over the Binary Symmetric Channel. We 
generalize the LP excess lemma to discrete, binary-input. Memoryless, Symmetric and LLR-Bounded 
(MSB) channels. As an application, we extend a result by the first author and H. Audah (IEEE Trans. 

Inf. Th., 2015) on the impact of redundant checks on LP decoding to discrete MSB channels. 

1 Introduction 

In 2003, Feldman 0 introduced Linear Programming (LP) decoding as a relaxation of Maximum Likeli¬ 
hood (ML) decoding. The good performance of LP decoding of LDPC codes and its relation to iterative 
decoding was established in multiple studies such as O El!4l El (a comprehensive survey is found in 0). 

The LP excess lemma was introduced and established in @ in the context of the Binary Symmetric 
Channel (BSC) as a technique to trade crossover probability for “LP excess” when analyzing the LP decoder 
error probability under the assumption that the all zeros codeword was transmitted. The lemma says that 
if the LP decoder works on a slightly nosier channel, we can guarantee that it corrects a slightly shifted- 
down version of the received LLRs. In dual terms, this implies the existence of a dual witness (4] where 
the variable nodes inequalities arc satisfied on the variable nodes with some constant positive “LP excess”. 
The lemma was used to study the LP decoding thresholds of spatially coupled codes (6]| and the impact of 
redundant parity checks on the LP decoding thresholds of LDPC codes on the BSC 0. 

In this paper we extend the LP excess lemma from the BSC to discrete, binary-input. Memoryless, 
Symmetric and LLR-Bounded (MSB) channels. We define the channel model in Section ITTT1 and we give 
the needed background on LP decoding in Section [L2l We state and prove our main result in Section [2] As 
an application, we use the extended lemma in Section[3]to extend the result of 0 to discrete MSB channels. 

1.1 Channel model 

We consider MSB channels: an MSB channel 0 is a binary-input Memoryless channel where the input 
alphabet is {0,1} and the transition probability has a Symmetry property as well as a Bounded LLR property. 
For simplicity of the presentation, we assume that the channel is discrete, i.e., the output alphabet £ is a 
finite set (or a countably infinite set). The channel is symmetric in the sense that we have a partition of £ 
into pairs (a, a*), such that Pr(o|0) = Pr(a*|l) and Pr(a|l) = Pr(a*|0). The pairing is a bijective map 
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* : £ —>• £ such that a** = a for each a E £. Thus the channel is fully specified by a triplet ch = (£,p, *), 
wherepis aprobability distribution on £ when 0 was transmitted, i.e., Pr(a|0) = p(a) and Pr(a|l) = p(a*). 
The Log-Likelihood-Ratios (LLR) L c h(-) = L(-) is a real-valued map on £ given by 


L(a) = In 


p(a) 
p(a *)’ 


Note that L(a) = — L(a*) for each a E £. We assume that the channel is LLR bounded in the sense 
that 11L ||oo is upper bounded by a constant. If £ is finite, LLR boundedness is equivalent to p(a) f 0 for 
all a E £. We denote by = // the LLR probability distribution given 0 is transmitted, i.e., // is the 
probability distribution of L(a) where a is sampled according to p. 

The importance of discrete MSB channels stems from the fact that they allow the decoder to use soft 
quantized information. They include for example the BSC, the mixed BSC-erasure channel and the finitely- 
quantized additive Gaussian-noise channel. The binary erasure channel is an example of a discrete symmet¬ 
ric channel with possibly infinite LLRs. 

We are interested in small distortions of discrete MSB channels: 


Definition 1.1 (Channel distortion). If ch = (£,p, *) is a discrete MSB channel and a > 0, we call 
channel ch! an a-distortion of ch if ch! = (£.//, *) for some probability distribution p' on £ such that the 
L i -distance 

lb ~p 'lit : = ~p '( a )I - a - 

a 

Note that, ch' shares with ch the same paring map *. 

For instance, consider the /3-BSC channel with cross over probability 3. An a-distortion of the /3-BSC 
is the p '-BSC where |/3 - /3'| < a/2. 

Notations. In this document we use a bold-faced notation to refer to n-dimensional vector: we transmit 
a length-n binary string x E {0,1}” and receive y E £ n of x. Additionally, we denote by p" the product 
distribution on £ n associated with p and //' the product distribution on M n associated with //. Thus, if 
x = 0, where 0 is the all-zeros vector, then y is distributed according to p n and the corresponding LLR 
vector 

7 = L(y) := (L( y ,))/ =1 E M" 

is distributed according to p n . 

1.2 LP decoding 

Let Q C Fg be an I^-linear code with blocklength n and ch = (£,p, *) a discrete MSB channel. Consider 
transmitting a codeword xeQ over ch, which outputs y E £". The ML decoder of Q is given by 

ML(y) = argmaxP Y |x(y|x). 

xeQ 

In terms of the LLR vector 7 = L(y), the ML decoder is given by 

ML q ( 7 ) = arg min(x, 7 ), 
xG Q 


where (x, 7 ) := Y,i x Hi- 

Feldman et al. [D O introduced the notion of LP decoding, which is based on relaxing the optimization 
problem on Q into a LP. Due to the linearity of the objective function (x, 7 ), optimizing over Q is equivalent 
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to optimizing over the convex polytope conv(Q) c M ra spanned by the convex combinations of the code¬ 
words in Q. The idea of Feldman is to relax coriv(Q) into a larger lower-complexity polytope. In general 
terms, an LP-relaxation of Q is a Q-symmetric convex polytope P C [0,1] '\ where Q-symmetry means that 
(|xj — y,|)' ; " = | € Q, for each x G Q and y G P JU. Note that Q-symmetry implies that Q C P. 

The LP decoder is given by 

LP p( 7) = argmin xg p(x, 7 ). 

While useful constructions of P are obtained from Tanner graph representations HI [H, it is simpler to 
establish the LP-excess lemma in the general framework of Q-symmetric poly topes P C [0, l] n . The Q- 
symmetry of P implies that when evaluating the LP decoding error probability, we can assume without loss 
of generality that the all-zeros codeword 0 was transmitted (TJ. Thus 7 ~ p n , where ft = is the LLR 
probability distribution given 0. As in previous works (TJEl, we assume that the LP decoder fails if 0 is not 
the unique optimal solution of the LP, i.e., the P-LP decoder succeeds on 7 iff LPpiy) = 0. 

We say that the LP decoder succeeds with LP excess £ on 7 if it succeeds on 7 —£1, i.e., LPp (7 —£1) = 
0, where 1 € M n is the all ones vector and (7 — £l)j = 7 , — £, for i = 1 ,..., n. 

For constructions of P from a Tanner graphs, LP excess can be interpreted in terms of the notion of a 
dual-witness H as follows. In dual terms, the P-LP decoder succeeds with LP excess £ on 7 iff 7 — £1 has a 
dual-witness, i.e., 7 has a dual-witness where each of the dual-witness variable nodes inequalities is satisfied 
with “LP excess £” (see Definition 2.1 and Theorem 2.2 in Q for the equivalent dual characterizations of 
LP decoding success). 

When studying the LP decoding error probability as the block length n tends to infinity, we consider an 
infinite family of F 2 -linear codes Q = {Q n }n and an associated infinite family of LP-relaxation V = {P n } n - 
We say that the P-LP decoder succeeds on ch with high probability if 

lim n _^oo Pt-y~/i™ [LP p n ( 7 ) 7 ^ 0 ] — 0 . 

We say that the P-LP decoder “succeeds on ch with LP excess £ with high probability” if 

lirn^oo Pr 7 ^n [LP Pn (7 - £1) ^ 0] = 0. 


2 LP excess lemma 

In this section, we extend the BSC LP excess lemma || 6 ] stated below to discrete MSB channels. 

Lemma 2.1 (@). (BSC LP Excess Lemma: trading crossover probability with LP excess) Consider the 
/3-BSC which crossover probability 0 < < 1/2. Let Q be an infinite family of¥ 2 -linear codes and V an 

associated family of LP-relaxations. 

Assume that there exists (3 < /3' < 1/2 such that the V-LP decoder succeeds on the ff -BSC with high 
probability. 

Then, there exists a £ > 0 such that the V-LP decoder succeeds on the f3-BSC with LP excess £ with high 
probability. 

Lemma 2.2. (MSB LP Excess Lemma: trading channel distortion with LP excess) Let ch be a discrete MSB 
channel, Q an infinite family of¥ 2 -linear codes and V an associated family of LP-relaxations. 

Assume that there exists a > 0 such that for each a-distortion ch! of ch, the V-LP decoder succeeds on ch' 
with high probability. 

Then, there exists £ > 0 such that the V-LP decoder succeeds on ch with LP excess £ with high probability. 
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In proving the Lemma, we follow similar steps to those taken in 0. The starting point in 0 is to 
realize the /T-BSC as a distortion of the /3-BSC resulting from the bit-wise OR of the /3-BSC error event 
with an independent Bernoulli random variable B. The distorted channel operates according to the original 
channel if B = 0 and it produces an error if B = 1. To generalize this construction, we use a similar 
Bernoulli-induced distortion of ch. The key new ingredient is a construction of a probability distribution q 
supported on the set of output symbols with negative LLRs. The distorted channel ch' operates according 
to the original channel if B = 0 and according to q if B = 1. A key property of the constructed q will be 
that the LLR map L' of ch' is a positive constant scale of that of ch, i.e., there exists a constant c € (0,1) 
such that L'(a ) = cL(a ) for all a G X. This property will be essential in extending the argument of 0 to 
our setup. 

Proof of Lemma IZ2l The proof is based on the fundamental cone. Let C n C R ra be the fundamental cone |[5j] 
of the P n - LP decoder, i.e., the set of all LLR vectors correctly decoded by the decoder: 

C7 n = { 7 €R":LP Pn ( 7 ) = 0}. 

Since LP p n ( 7 — £1) = 0 is equivalent to 7 £ C n + £1, our objective is to show that there exists a £ > 0 
such that /x”(C' n + £1) = 1 — o n (l). The hypothesis of the theorem guarantees that for any a-distortion ch’ 
of ch, ij' n (C n ) = 1 — o n (l), where p! = is the LLR probability distribution of ch! given 0. 

By the definition of the LP decoder, C n is the interior of the polar cone of P n , i.e., 

Cn = { 7 iE R n : ( 7 , x) > 0 for each nonzero x £ P n }- 

We note that since P n C [0,1]" C (R + ) n , C n is closed under translation by vectors in the non-negative 
quadrant, i.e., C n + (R + ) n C C n . We will argue that £ exists using only the property that C n C R n is a 
convex cone such that C n + (R + ) n C C n . 

Consider the partition of X into three sets: 

X_ = {a € X : p{a ) < p(a*)} 

X 0 = {a € X : p(a) = p{a*)} 

X+ = XI. 

Thus L is negative on X_, zero on Xo and positive on X + . Without loss of generality, we assume that X_ 
and X + are nonempty (otherwise, the channel capacity is zero). 

Let 0 < 6 < 1 be a constant such that 5 < a/2 and define channel ch! = (X,p', *), where p’ is the 
distribution on X given by 


p'(a ) = 5q(a) + (1 — S)p(a) if a € X_ 

p'(a ) = (1 — S)p(a) if a € Xq U X + , 


and where q is a probability distribution on X_ that will be specified later. We will sample from // as 
follows. First we sample a Bernoulli random variable B ~ Ber(S) which takes the value 1 with probability 
5. If B = 0, we sample from p and if B = 1, we sample from q. Channel ch' is an a-distortion of ch 
because \\p — p'\\i <25 < a. The LLR map of ch' denoted by L' is given by: 


L'(a) = In 


5g(a) + (1 - 5)p(a) 
(1 — 5)p(a*) 


if a € X_, 
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L'(a) = —L'(a*) if a £ X + and L'(a) = 0 if a £ So- We choose q so that there exists a constant c £ (0,1) 
such that L’(a) = cL(a) for all a £ £ which is guaranteed by enforcing L\a) = cL(a ) on a £ £_, i.e., 


5q{a) + (1 - S)p(a) _ f p{a) V 
(1 — 5)p(a*) \p(a*)) ’ 

Solving for q(-), we get 

q(a) = ^—r^p{a) 


pifl*\ 

P(a) 


1—C 


- 1 


a £ £_. 


a € £_. 


Since c £ (0,1) and p(a*) > p(a) for a £ £_, we have g(a) > 0 on a £ £_. To guarantee that 
Yla ?( a ) = 1> we choose c £ (0,1) so that s(c) = where 



This follows from the continuity of s(-) as a function of c and the facts that s(l) = 0 and 

a(°) = p ~ p = P( S +) ~ P( S ~) > °- 

aeE- agS_ 


since S + and X_ are assumed to be nonempty. In what follows, fix <5 £ (0,1) to be any constant such that 
5 < j such that < p(^ + ) ~ p(^~) 1:0 guarantee the existence of q and c. 

In the remainder of the proof we follow the steps in Q: we use an averaging argument followed by 
Markov Inequality. For clarity, we will use capital letters to refer to random quantities. Define f : Y7 1 x 

£ n x {0, l} n -»■ £ n by 


f(y,z;b)* 


Z i if bj = 1 

y i if bj = 0. 


Thus, if Y ~ //', Z ~ q n and B ~ Ber(5) n , then f(Y, Z; B) is distributed according to //", and = 
L/(f (Y, Z; B)) is according to ///". For each y £ E n , define the random vector 


r'(y, Z; B) = (5 L'(f (y, Z; B)) = /3c L(f(y, Z; B)) £ R n 


over the random choice of Z ~ q n and B ~ Ber(5) n , where /3 > 0 is a constant to be specified later. 
Denoting by 1 c n : K n —> {0,1} the indicator function of C n (i.e. lc n (l) = 1 iff 7 £ C n ), we define 
w(y) £ M n for y £ T, n by 

w(y) = E z ,b [r'(y, Z; B) x l Cn (r'(y, Z; B))] . (1) 

For each y £ X n we have w(y) £ C n since C n is a convex cone. Thus, interpreting vector inequalities 
coordinate-wise, 

p n [C n + £1) > Pr Y ^ p n [(L(Y) - w(Y)) > £1] (2) 

because v > w(y), for any v £ C n and any y £ S n since C n + (R + ) n C C n . Equation can be written 
as 


w (y) = E [T'(y, Z; B)] - E [T'(y, Z; B)|^(y, Z; B)] • d»(y), 
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where </?(y, Z, B) is the error event ‘T / (y, Z; B) ^ C n ” and 


$(y) := Prz,B[^(y; z ^ B )]- 


The first term 

where 


E [r'(y, Z; B)j] = /3c( 1 - S)L( y,) - /Ms, 


s:=-E^ ? [L(Z)] 

is a positive scalar because q is supported on X_ and E z~ q \L{Z)] is strictly negative. The second term 

E [r / (y, Z; B)j|</?(y, Z; B)] > -f5\\L '\\= -^cH^Hoo 
since the LLRs are bounded. It follows that 


w(y)j < /3c[(1 - <5)T(yi) - 5s + ||L|| 00 $(y)]. 

Setting we get 

w( y ).<. (y .)- fa -»y< y ) 

Therefore, to guarantee that the vector inequality L(y) — w(y) > £1 holds, it is enough to require 
the scalar inequality 5s — ||L|| 00 < I > (y) > £(1 — 5). Note this reduction of the vector inequality to a scalar 
inequality critically depends on the choice of q so that L' = cL. Setting £ = , we get from ([2]) that 


l-Ai n (C' n + £l)<Pr Y $(Y)> 


8s 

2||L||cx> 


Using Markov Inequality, and the fact that Ey I‘NY)] = 1 — n' n (C n ), we obtain 

i - n n (c n + a) < ^(i - v ,n (c n )). 

Since n' n (C n ) = 1 — o n (l), we conclude that [i n (C n + £1) = 1 — o n (l), where £ > 0 is constant which 
depends on a and the channel ch. □ 


Remark 2.3. I) If we replace probability distributions with densities, the LP excess lemma and its proof 
hold for continuous MSB channels. 


II) We conjecture that the LLR boundedness is not needed for the lemma to hold. One justification of this 
conjecture is the Gaussian channel discussed below. 


2.1 Gaussian channel 

On the er-Additive White Gaussian Noise (cr-AWGN) channel, we receive Y = (— l) x + aZ, where x — 0 
or 1 is the transmitted bit and Z ~ J\f (0,1), the standard Gaussian distribution. The AWGN has unbounded 
LLRs. 

By a simple scaling argument, the following version of the LP excess lemma holds on the AWGN: 

Lemma 2.4. Let Q n C F 2 be an ¥ 2 -1 inear code, P n C M n an LP-relaxation of Q n and a' > a > 0. The 
probability of success of the P n -LP decoder on the o’-AWGN is equal to its probability of success on the 
a-AWGN with LP excess f where ^ = a . 


6 










Proof. The LLR map is L(y) = \y (e.g., |[T1). Assume that 0 was transmitted and let // and p! be the LLR 
densities associated with cr and o', respectively. Since 

— (1 + o'z) = 1 + oz — £, 
o 

we get p' n (C n ) = p n (C n + £1), for each C n C M n closed under multiplication by positive scalars and in 
particular for the fundamental cone C n of the P n -LP decoder. □ 

The distinguishing features of the AWGN from other channels in this context are: (1) scaling Z corre¬ 
sponds to distorting the channel and (2) the LLR map is linear in y. 

3 Application to redundant parity checks 

The BSC LP excess lemma was used in 0 to show that the LP decoding threshold of LDPC codes on 
the BSC remains the same upon adding all redundant parity checks, assuming that the underlying Tanner 
graph has bounded degree and possesses two natural properties called asymptotic strength and rigidity (see 
Corollary 1.7 in 0). One implication of this result is that the BSC threshold is a function of the dual code 
and is not tied to the particular Tanner graph realization of the code. We use in this section our extension of 
the LP excess lemma to extend the result of 0 from the BSC to discrete MSB channels: 

Theorem 3.1. Let Q = {G n }„ be an infinite family of Tanner graphs, where G n has n variable nodes. 
Let Q = {G n } n be the resulting family of Tanner graphs obtained by adding all redundant checks, i.e., the 
parity check nodes of G n correspond to all the nonzero elements of the dual code of G n . Assume that Q 
has bounded check degree and that Q is asymptotically strong and rigid. Let ch be a discrete MSB channel. 
Assume that there exists a > 0 such that for each a-distortion ch' of ch, the Q-LP decoder succeeds on ch' 
with high probability. Then, the Q-LP decoder succeeds on ch with high probability. 

In order to prove the theorem we only need the following extension of Theorem 1.2 in 0 to discrete 
MSB channels: 

Lemma 3.2. Let Q , Q, ch, a, ch' be as in Tlieorem \3.1\ and Let d be the maximum degree of a check node in 

_ ^ ^ 

Q. For k > d, let Q := { G n } n be the resulting family of Tanner graphs obtained by including all redundant 
checks of degree at most k. There exists a sufficiently large constant k > d -where k depends on a and the 

_ k 

channel only- such that the Q -LP decoder succeeds on ch with high probability. 

Proof of Theorem [7771 Following the proof of Corollary 1.7 in 0, Theorem 13.11 follows from Lemma lT2l 
and the rigidity of Q which implies that for each constant k > d, the LP decoding polytope P(G n ) = P(G n ) 
for n large enough. □ 

Proof of Lemma \3?2\ We use below the terminology of the proof Theorem 1.2 in 0 to explain the needed 
modifications. At a high level, the following changes are needed: 

• Instead of a variable received correctly or in error, we have positive or nonpositive LLRs respectively. 

• The value of LP excess is £ instead of |. 

• The maximum absolute value of a received LLR is the constant H-LHoo instead of 1. 
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More specifically, consider operating the G n - LP decoder on ch : assume that the all-zeros codeword was 
transmitted and consider the received LLR vector 7 ~ p™ h . By the LP excess lemma, there exists a constant 
£ > 0 (dependent on a) such that with high probability, the G n -LP decoder corrects 7 with LP excess £, 
i.e., it corrects 7 — £1. In what follows, consider any such 7 £ R n . To verify Lemma lT2l we will show that 

_fc 

the G„-LP decoder corrects 7 for a sufficiently large constant k > d which depends on £ and the channel 
(and does not depend on n). For notational simplicity, we will denote G n , G n and G n by G, G and G, 
respectively. Also, let E , E k and E be the set of edges of G, G k and G, respectively. 

By Theorem 2.2 in Q, there is a hyperflow w : E — > R in G for 7 — £1. Hence, 

F (w) < 7-fl. 

where F (w) £ M n is the flow as specified in Definition 2.1 in 10. Let 

V + = 


and 

= {i : 7i - f < °} 

be the set of variables nodes with positive and nonpositive “shifted LLR” respectively. Since G contains 
all redundant checks, we can assume by Lemma 4.2 in Q that w is primitive, hence the inflow to each 
variable in V + is zero and the outflow from each variable in V~ is zero. Following (7|, define the trimmed 
hyperflow and the resulting risky and problematic variables as follows. Trim w by removing all check nodes 
of degree larger than k. The trimming process leads to a distorted dual witness w k : E k —y K in d. 
The problematic variables nodes are those for which the hyperflow variables nodes inequalities of w k are 
violated with respect to 7 . A variable node is called risky if it receives at least | flow from the removed 
check nodes, thus all the problematic variables are risky. The set of risky variable nodes is called U. We 
have U C V~ since w is primitive. Hence 

F i(w k ) <0 if i £ U, and 

Fflw k ) < 7i -£/2 iff £U. 


Since all the removed checks have degree larger than k and since < 11 A [| ^ for each z, the removed checks 
give the variables in V~ at most 

k — 1 ~ k — 1 


flow. It follows that 

ITJI ^ 11 L 11 00 

1 1 - - 1 )' 

Since w is primitive, to fix w k on the problematic variables, it is enough to give each variable in U an ||Lj|oc 
flow. Following Q, we do that by exploiting the asymptotic strength of Q and the remaining excess on the 
nonrisky variable nodes. The remaining LP excess on each nonrisky variable is at least £ — | = Consider 
the asymmetric LLR vector r £ M n given by: 


L||oo if i £ U 
otherwise. 

We use the remaining excess to fix w k by superposing w k 



with a dual witness for r. 





Since Q is asymptotically strong, there exists a constant 6 > 0 dependent on — such that if \U\ < 5n, 
the LP decoder of G succeeds on , 177 !— and hence on r. Thus, if 


2IILII 


- f) 


<<5, 


then r has a dual witness v : E —> R in G. Since k > d, let v k : E k —> R be the extension of v to G k by 
zeros. Thus F (■ v k ) < t and accordingly 


F {w k + v k ) < 7 . 

Therefore, w k + v k is the desired dual witness of 7 in G k . It follows (from Theorem 2.2 in Q) that the 

_ fc 

G -LP decoder successfully corrects 7 . 

In summary, there exists a constant 5 > 0 dependent on — such that if 


k = max 





_ fc 

which depends on the £ and the channel, then the G -LP decoder corrects 7 for any 7 <G M n such that the 
G-LP decoder corrects 7 — £1. □ 

Note that the proof of Lemma [3721 breaks down if the LLRs are unbounded even if Lemma lT2l holds for 
channels with unbounded LLRs (see Remark [iOl II). 
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